Multi-objective optimisation of relevance vector machines: selecting sparse features for face verification ICML/UAI/COLT 2008 Workshop on Sparse Optimisation and Variable Selection

The relevance vector machine (RVM) (Tipping, 2001) encapsulates a sparse probabilistic model for machine learning tasks. Like support vector machines, use of the kernel trick allows modelling in high dimensional feature spaces to be achieved at low computational cost. However, sparsity is controlled not just by the automatic relevance determination (ARD) prior but also by the choice of basis functions or, equivalently, kernels. In particular severe over-fitting occurs when multi-resolution kernels, such as wavelets, are employed. Also, the lack of control over the weight variances means that overfitting can occur due to the greediness of feature selection. For regression this has been combated through the use of a smoothness prior (Schmolck and Everson, 2007) but the methodology does not carry over to classification problems. Additionally, it is often unclear what the costs of misclassification are and commonly one wants to assess performance over a range of misclassification costs.